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Abstract 

The Ontario Ministry of Education and Training’s Task Force on University 
Accountability first proposed key performance indicators (KPIs) for colleges 
and universities in Ontario in the early 1990s. The three main KPIs for Ontar¬ 
io universities are the rates of (1) graduation, (2) employment, and (3) Ontar¬ 
io Student Assistance Program loan default. This exploratory and descriptive 
study examined the perceptions of 12 key informants from 11 participating 
universities about the efficacy and effectiveness of these KPIs. The results of 
this study demonstrate that a clear majority of participants believe these KPIs 
are not having the intended impact. This paper analyzes the evidence and 
makes recommendations designed to foster efficient collaboration between 
stakeholders; it also asks all parties to clarify their goals, agreed expectations, 
and requirements, in order to develop effective measures of institutional per¬ 
formance and accountability and address the political needs of the govern¬ 
ment, the universities, and the public. 

Resume 

Au debut des annees 90, le Groupe de travail sur la responsabilisation 
des universites du ministere de l’Education et de la Formation de l’Ontario 
a d’abord propose des indicateurs de performance pour les colleges et les 
universites de l’Ontario. Ces trois principaux indicateurs cles de performance 
(ICP) pour les universites ontariennes sont (1) les taux d’obtention des 
diplomes, (2) les taux d’embauche et (3) les taux de defaut de remboursement 
des prets du Regime d’aide financiere aux etudiantes et etudiants de l’Ontario 
(RAFEO). Cette etude exploratoire et descriptive a examine la perception 
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de 12 repondants cles (provenant de 11 universites participantes) envers 
l’efficacite de ces ICP. Les resultats de cette etude demontrent qu’en majorite 
absolue, les participants croient que ces ICP n’ont pas l’effet envisage. Ce 
rapport analyse les preuves et emet des recommandations ayant pour but de 
favoriser une collaboration efficace entre les intervenants. II demande aussi a 
chaque partie de clarifier les objectifs, les attentes convenues et les exigences 
etablies dans le but d’elaborer des mesures efficaces de la performance et 
de la responsabilisation des etablissements, ainsi que d’adresser les besoins 
politiques du gouvernement, des universites et du public. 

Introduction 

The Ministry of Education and Training’s Task Force on University Accountability 
first proposed performance indicators for colleges and universities in Ontario in the ear¬ 
ly 1990s (Ministry of Education and Training, 1993). The decision to develop and use 
key performance indicators (KPIs) was made that same decade, to help students reach 
informed program decisions (MET, 1998). The required KPIs for the Ontario universi¬ 
ties were graduation rates, employment rates, and Ontario Student Assistance Program 
(OSAP) loan default rates. The research project behind this paper studied the impact of 
KPIs on a large group of Ontario universities—specifically, the efficacy of KPIs in the con¬ 
text for which they were adopted, 10 years following their implementation, as “key infor¬ 
mants” saw them. This investigation further sought to identify areas of concern among 
the participants and to solicit suggestions for either altering or improving the KPIs’ ef¬ 
fectiveness. The informants from the Ministry of Education and Training’s Task Force 
on University Accountability, known as the Broadhurst Task Force (named after the task 
force’s chair, Dr. William Broadhurst), included a member appointed by the ministry, a 
Council of Ontario Universities (COU) representative, and an observer/consultant of the 
task force. This paper presents a report based on the responses to online survey state¬ 
ments and one-on-one follow-up interviews with key informants from the participating 
universities. Collectively, the results provide a measure of the strength of the respondents’ 
agreement or disagreement with a series of statements on a four-point, forced-choice re¬ 
sponse scale (from strongly disagree to strongly agree). 

The Ministry of Training, Colleges and Universities (MTCU) has declared that the KPIs 
are published to enable parents and students to make an informed choice of program and 
institution for the student’s post-secondary education, to help universities evaluate the 
quality of programs, and to improve the universities’ performance (MTCU, 2012). Having 
examined the history of why and how KPIs were introduced in Ontario, I was motivated 
to research the following lines of inquiry: (i) to assess whether these KPIs are having the 
intended effect and (ii) to reflect on whether changes could improve their effectiveness. 
As this paper will demonstrate, the findings of the study provide a better understanding of 
the impact of the KPIs, as well as suggestions for alternative models and/or adjustments 
to improve future performance. The collected data may also be valuable resources to in¬ 
form evidence-based MTCU policy decisions that will directly affect Ontario’s universities 
and their stakeholders. If the KPIs were initially effective gauges after the collection of 
KPIs was imposed in September 1999, are they still? Is there a need to re-examine other 
accountability measures that may be more effective in the current and future context for 
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Ontario universities? Addressing these general research questions, this paper is part of 
a larger study driven by seven research questions, of which I consider the following four 
for discussion here; these share a common theme and were selected for their relevance to 
Ontario universities. Based on the observation and experience of key informants: 

1. To what extent does each of the KPIs impact institutional planning in participating 
Ontario universities? 

2. To what extent do universities use KPI results for recruiting students, setting tu¬ 
ition fees for specific programs, or other purposes? 

3. What is the effectiveness of KPIs in improving the performance of universities at 
the institutional and program levels? That is, what is the impact of each of the KPIs, 
if any, on academic planning and delivery, according to this group of stakeholders? 

4. How are the three KPIs currently in place appropriate measures of institutional 
performance and accountability? What other indicators may better serve all stake¬ 
holders—i.e., the government, the universities, and the public? 

Background and Literature Review 

Since the 1960s, Ontario universities have relied on government grants for the ma¬ 
jority of their institutional revenue, and the universities have been accountable for the 
expenditures of these funds. This accountability involves multiple groups—i.e., university 
governing boards, faculty and students, government, and taxpayers. Specifically, Ontario 
universities were required to file externally audited financial reports to the government. 
Monahan (2004) observes that, based on the rules around eligibility for receipt of operat¬ 
ing grants and capital funding, these requirements satisfied the need to monitor account¬ 
ability. However, in the early 1980s, the provincial auditor questioned the adequacy of 
these simple requirements. A decade later and subsequent to three inspection audits at 
Trent University, the University of Guelph, and the University of Toronto in 1988-1990, 
the Ontario auditor concluded there was inadequate accountability for university funding 
(Guelph, 1992). The decision was influenced by a CCAF (Canadian Comprehensive Audit¬ 
ing Foundation) paper, which quoted a statement made by one of Canada’s leading edu¬ 
cational experts that they (the universities) had to, and wanted to, be more accountable; 
they needed defined indicators to measure performance (CCAF, 1988). 

Gaither, Nedwek, and Neal (1994) observed that policy makers and the academic com¬ 
munity explored using a system of indicators to raise warning signs about the efficiency 
and effectiveness of higher education. Outcomes, results, and performance thus would 
determine funding allocation. Depending on the results desired, the role of performance 
indicators could vary from a minor role as a supporting tool to an expanded, crystallized 
role to measure results and thereby satisfy the public appetite (Gaither et al., 1994). In 
1997, Queen’s University defined a performance indicator as any parameter, measure, or 
quantitative instrument intended or used for: (i) evaluating, measuring, ranking, or com¬ 
paring the performance, productivity, or standards of an academic unit, program, faculty, 
school, or institute, in whole or in part; and (ii) making decisions that affect the terms or 
conditions of employment for a faculty member (QUFA, 1997). Burke (1997) compared in¬ 
dicators of performance-funding programs in eight states in the United States, observing 
that indicators among institutions differed in number, content, and terminology. McMil- 
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lan and Datta (1998) claimed that data envelopment analysis (DEA) is more reliable than 
performance indicators in terms of comparisons. DEA includes output variables such as 
the number of graduate enrolments in different degree-level programs, total sponsored 
research expenditures, and input variables such as total number of full-time faculty mem¬ 
bers eligible for Tri-council or Canada Council grants (McMillan & Datta, 1998). However, 
the Ontario government did not include these among the KPIs that it ultimately selected. 

Frink and Klimoski (2004) claimed that shared expectations are important for an or¬ 
ganization’s response regarding its members’ accountability. In order to anticipate and 
model agent behaviour effectively, it is essential to determine the nature and clarity of the 
expectations of members and of the role set, and the degree of alignment with these ex¬ 
pectations (Frink and Klimoski, 1998). Ansell and Gash (2007) agreed that collaborative 
governance brings public and private stakeholders together in common fora to engage in 
consensus-oriented decision making. Frink and Klimoski (2004) also stated that shared 
expectations are important for an organizational response to accountability. Similarly, 
Hernandez and Hodges (2001) stressed the necessity of seeking all stakeholders’ input. 
Ultimately, there remains a need both to clarity and to satisfy the goals and expecta¬ 
tions of the ministry and the universities in order to improve KPI-related institutional 
accountability processes. In this respect, the observations of Knouse (1979) remain valid: 
accountability requires the clarification of role expectations by the parties involved; both 
parties should be clear on each other’s goals and expectations. As Burke and Minassians 
(2003) appropriately recommended for the American system, the Ontario government 
and universities should work together to develop a set of indicators that incorporate both 
quality and efficiency indicators in a way that responds to the concerns of both parties. 

Lang (2013) commented that performance funding should be tied to quality, signifi¬ 
cance, and influence—that is, “the difference that performance makes” (Lang, 2013, p. 
15). He also added that the “one size fits all” homogenizing effect of performance indica¬ 
tors impaired the essence of performance funding. The incapability of the current perfor¬ 
mance indicators in measuring quality has led the commercial surveys and rankings of 
universities to substitute reputation for quality (Lang, 2013). In Ontario, the performance 
funding program tied with performance indicators has changed four times. These indi¬ 
cators do not capture each university’s total performance, and other sources of funding 
for universities are significantly more than the current performance funding program, 
resulting in diminished reliability of both performance measurement and the influence of 
performance funding (Lang, 2013). 

According to Salmi (2009), with respect to characteristics indirectly related to re¬ 
search, a “world-class” university (as defined by national and international rankings 
published by third-party organizations) must have (i) a sound financial base and (ii) a 
diversified source of income, including research income; (iii) provide a high-quality and 
supportive research environment (e.g., high-quality buildings and facilities); and (iv) have 
a first-class management team with strategic vision and implementation plans. Although 
the KPI initiative does not claim explicitly to identify “world-class” universities, nor are 
KPIs intended overtly to rank universities, their data may encourage ranking. There are 
currently a few well-known national and international ranking publications (e.g., the Ma¬ 
clean’s University Rankings and the Times Higher Education Supplement ) that claim 
to report performance measures. In the case of Maclean’s, the methodology has changed 
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from originally using a single set of indicators for all universities to currently using three 
separated ranking groups: namely “medical doctoral,” “comprehensive,” and “primar¬ 
ily undergraduate.” Issues of data integrity, differentiation of universities’ missions and 
characteristics, and differences in students’ preferences in the Maclean’s ranking system 
jeopardize the accuracy and relevance of the rankings (Smith, 2000). For example, the 
Maclean’s University Rankings do not consider the KPIs; instead, Maclean’s uses mea¬ 
sures that are distinctively different from KPIs. 

However, the stakes for universities that may struggle to meet MTCU’s KPIs are high. 
MTCU has created a benchmark (on benchmarking, see Alstete, 1995, p. iii) for each of 
the three indicators of graduate employment rate six months after graduation, gradu¬ 
ate employment rate two years after graduation, and graduation rates for calculating the 
distribution of funds to universities. The MTCU set the benchmark for the particular in¬ 
dicator at a certain percentage below the system average; institutions that fall below the 
benchmark do not receive funding (MTCU, 2009). In 2000, MTCU introduced perfor¬ 
mance-based funding for colleges and universities, allocating $16.5 million to universities 
based on graduation rates, employment rates six months after graduation, and employ¬ 
ment rates two years after graduation (MTCU, 2001). 

Research Design 

This study explored the efficacy of the three KPIs for Ontario universities over a 10- 
year period (1993-2012). The perceptions of key informants at participating universities 
and an analysis of KPI reports from 1993 to 2012 provide a deeper understanding of the 
efficacy of the KPIs in driving performance improvements at these universities during the 
period of study. In evaluating these data, this study used a sequential mixed methods, and 
exploratory descriptive research design. 

To answer the research questions, I collected data in four phases. The first phase fo¬ 
cused on document analysis as well as interviews with individuals who had participated in 
the initial development of the KPIs; the latter included interviews with three members of 
the Broadhurst Task Force. In the second phase, documents reporting KPI performance 
were analyzed and trends were identified. The primary research sources were publicly 
available mission statements and strategic plans of all Ontario universities, as well as 
MTCU policies related to KPIs, to see how they were related to the KPI findings. Addi¬ 
tional sources for this phase were published data on graduation rates, employment rates, 
and OSAP default rates for 1998-2010, provided by MTCU. Two officials from the Post¬ 
secondary Accountability Branch of MTCU provided further primary evidence on KPI 
performance and trends. Phase three of the project invited 20 universities in Ontario to 
participate in an online survey. The universities covered the spectrum from primarily un¬ 
dergraduate to comprehensive and medical/doctoral institutions. Eleven universities ac¬ 
cepted the invitation to participate in this study (a 55% survey response rate). Participat¬ 
ing university presidents identified each of the key informants. The latter were preferred, 
in this study, to other academic bodies, including faculty and students, for their greater 
authority over policy and administrative decisions, their inside knowledge of data related 
to KPIs in their respective institutes, and their interest in leveraging this knowledge to 
attract more funding. The sample of participating universities was fairly representative of 
the total population, based on size, urban/rural location, and programming. The survey 
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structure centred on the themes identified in the data collected in the first two phases. 
The selection criteria for the invited participants/respondents of this study were their di¬ 
rect roles in the areas of institutional planning, budgeting, and institutional performance, 
which would enable them to provide a perspective from the institutional level. The survey 
posed questions for evaluation on a four-point response scale, with the option of providing 
additional comments. The fourth phase of this project explored further themes generated 
by this survey via one-on-one interviews with seven key informants from the university 
respondents. Further interviews with two officials from MTCU, mentioned above, also 
provided a more comprehensive perspective on how the ministry perceives and uses the 
data. Overall, 17 key informants contributed to this study: 12 from universities; two from 
MTCU, selected by the Minister; and three from the Ministry of Education and Training’s 
Task Force on University Accountability. 

Usefulness of KPIs for Institutional Planning 

Most of the university respondents (10 of 11) agreed that the impact of the “gradua¬ 
tion rates” KPI on their universities’ institutional planning is high (Table 1). Six and eight 
of the 11 respondents, respectively, disagreed that the impact of the “employment rate” 
and “OSAP loan default rate” KPIs is high with respect to their universities’ institutional 
planning. Since the majority of the respondents either disagreed or strongly disagreed 
that the OSAP default rate informs institutional planning, a follow-up interview question 
investigated why they did not perceive the OSAP default rate to be helpful to universities. 
Of the seven university participants interviewed, a majority (five of the seven) stated that 
other factors beyond the institution’s control impact these KPIs, such as employment op¬ 
portunities, labour market needs, and the state of the economy. 

Table 1. 

Percentages of Positive Responses (Agree and Strongly Agree) from Online Survey on 
KPIs, their Impact, Usefulness, and Effectiveness 



Graduation Rates 

Employment 

Rates 

OSAP Loan 
Default Rates 

High impact on institutional planning 

91% 

(10 out of 11) 

45% 

(5 out of 11) 

20% 

(2 out of 10) 

Used in student recruitment 

73% 

(8 out of 11) 

45% 

(5 out of 11) 

9% 

(1 out of 11) 

Used in setting tuition fees for specific 
programs 

0 % 

(0 out of 11) 

18% 

(2 out of 11) 

0% 

(0 out of 11) 

Used for other purposes 

100% 

(11 out of 11) 

82% 

(9 out of 11) 

40% 

(4 out of 10) 

An effective factor in improving institu¬ 
tional performance 

73% 

(8 out of 11) 

55% 

(6 out of 11) 

0% 

(0 out of 10) 

An effective factor in improving perfor¬ 
mance at program level 

80% 

(8 out of 10) 

20% 

(2 out of 10) 

0 % 

(0 out of 10) 


Source: Responses from the online survey provided by universities’ key informants. 
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In reference to the funding associated with KPIs and its impact, one of the respon¬ 
dents said that “based on the relatively small amount ($23 million performance funding), 
nobody likes them, nobody pays attention to them or uses them directly or very much 
in their programming [evaluation] processes.” The same person also stated that due to 
changes in the funding allocation formula, there are “fewer true winners and losers,” 
as the threshold point of the funding rewarded is relatively minor. Another respondent 
raised the point that the university environment is so complex that having three indica¬ 
tors is not formative enough to direct an institution. 

While most participating universities use graduation rates to a limited extent for insti¬ 
tutional planning, the universities use neither employment rates nor OSAP default rates 
during institutional planning processes to any considerable degree. One respondent com¬ 
mented that the approach of pooled funding for KPI performance hinders KPI usefulness 
in institutional planning, since most of the decisions that would impact program excel¬ 
lence are made at the program level. 

For internal purposes, all of the participating universities reported that instead of 
the KPIs, they use other institution-specific and “plan-focused” indicators, which enable 
drill-down analysis related to strategic priorities that each university sets in its academic 
plan. The advantage of institution-specific performance indicators, as opposed to On¬ 
tario’s KPIs, is that the former are more helpful for managing performance and planning, 
more relevant to measuring outcomes, outputs, and impact, and more useful for evalua¬ 
tion and communication purposes. One of the university participants mentioned that his/ 
her institution used more than 100 indicators for different purposes. Interestingly, most 
of these internal indicators reported by the survey participants are similar to those in the 
list of indicators that the CUPA (Council on University Planning and Analysis) Commit¬ 
tee on Accountability, Performance Indicators and Outcomes Assessment initially rec¬ 
ommended to the Broadhurst Task Force (CUPA, 1993). Hernandez and Hodges (2001) 
have stressed that clear linkages between identified goals, strategies, and implementation 
plans are essential to inform strategic planning. The gulf between the scope of the KPIs 
and institution-specific approaches to measuring performance suggests that these link¬ 
ages between KPIs and the institutions affected by them are not strong. 

Usefulness of KPIs for Student Recruitment 

Among MTCU respondents, there was a perception that although universities were 
using KPIs in their active student recruitment, they did not know the impact of KPIs on 
student preferences. In contrast, a majority of university respondents (eight) agreed that 
graduation rates are useful data for their institutions with respect to student recruitment 
efforts; however, they disagreed (six and 10 out of 11, respectively) with the proposition 
that employment rates and OSAP loan default rates serve the same purpose (Table 1). 
Most key informants commented in the follow-up interview that the KPIs are of limited 
use to student recruitment. While KPIs are useful as a public relations tool in two inter¬ 
viewees’ institutions, this is obviously only the case when the KPI results are favourable. 
In general, respondents indicated that most participants’ universities use graduation rates 
for student recruitment but do not use employment rates or OSAP default rates widely for 
that purpose. One of the university participants stated that in his/her opinion, the major 
factors influencing students’ decisions are program availability, quality, reputation, and 
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location. Given that the initial purpose of the KPIs was to inform the public—particularly 
so as to assist parents and prospective students in selecting preferred, appropriate insti¬ 
tutions for post-secondary studies—this project conducted a review of student recruit¬ 
ment-related online resources for 20 universities to identity whether these sites included 
information on KPI performance. None of these resources included reports on KPIs or 
web-links related to KPIs for any of their online recruitment efforts, indicating that the 
importance of KPIs as a tool for current recruitment campaigns is, at best, minimal. 

Usefulness of KPIs for Setting Tuition Fees for Specific Programs 

With respect to the usefulness of KPIs in setting tuition fees for specific programs, all 
11 university participants disagreed with the proposition that the graduation and OSAP 
default rates were useful. Furthermore, only two respondents (18%) agreed that em¬ 
ployment rates are one of the factors considered by their respective institutions when 
determining tuition fees for specific programs (Table 1). In follow-up interviews, the 
respondents explained that the province regulates tuition fees. Specifically, MTCU has 
regulated the Tuition Fee Policy for publicly assisted universities since 2006 (MTCU, 
2010). However, universities were able to impose tuition increases during the inception 
of KPIs. Furthermore, four of the seven interviewed respondents felt that these KPIs were 
disconnected from setting tuition fees for specific programs. They report that high-level 
indicators, such as employment rates, cannot translate or properly reflect students’ ability 
to find employment in relation to specific programs. In the respondents’ views, because 
students often do not pursue careers within disciplines most closely related to the field 
of their degree and/or program, the employment rate KPI is of limited use. For example, 
one respondent stressed that “you offer a [university] program, [designed] not to equip 
individuals for the workplace, but to equip them for life, so it’s not as if you are going to 
tie directly into some sort of employability or retention data.” Another respondent com¬ 
mented that the typical delay in reporting employment data about the labour market (five 
years) limits the usefulness of employability statistics. Both the scope and the applicabil¬ 
ity of data were concerns—for example, that “the more granular the level of planning for 
specific programs is, the less it is useful to look at the global employment and graduation 
rates.” Finally, participants’ institutions did not use OSAP default rates at all. Only two 
participants reported that their universities used employment rates for setting tuition 
fees, and then only for very specific programs; they viewed OSAP loan default rates as 
irrelevant for this purpose. Stressing that the province regulates tuition fees, most key 
informants stated that the KPIs are “disconnected” from their control. 

Usefulness of KPIs for Other Purposes 

All respondents agreed that graduation rates are useful for other purposes, and 82% 
of the respondents (i.e., nine) agreed that employment rates are also useful for other 
purposes. However, only four agreed that OSAP loan default rates are useful for other 
purposes in their institutions (Table 1). In the comment section of the survey, three re¬ 
spondents clarified that they use the KPIs as content for “communication messages” and 
“background data scans to support the review of existing and new programs.” 
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Usefulness of KPIs for Improving Performance 

With respect to improving institutional performance, eight and six of the respondents, 
respectively, agreed that graduation and employment rates are effective factors in improv¬ 
ing institutional performance (Table 1). However, none felt the OSAP loan default rates 
were useful in this regard. At the program level, a majority of respondents (80%) agreed 
that graduation rates have an impact on performance improvement (Table 1). However, 
they did not view either employment or OSAP loan default rates as impactful: only two 
out of 10 respondents agreed that employment rates affected performance improvement 
efforts, and none agreed with this statement with respect to OSAP default rates (Table 1). 
None of the respondents felt that the OSAP default rates have any impact on improving 
performance at the institutional or program level. As identified earlier, this is due, in the 
respondents’ opinions, to factors beyond the institutions’ control. Thus, there is a percep¬ 
tion that this KPI is “disconnected” from institutional and program performance. 

Based on the input from the university participants surveyed and interviewed, the ef¬ 
ficacy of the KPIs as a means to affect university behaviour and performance funding is 
minimal at best, and then only for very different purposes than the initial accountability 
goals. The participants recognized some usefulness of graduation and employment rates 
for communication and recruitment purposes. However, the participants were concerned 
about what they viewed as a disconnect between the KPI data, the universities’ mission, 
and universities’ ability to impact these outcomes. Further, respondents also expressed 
reservations about what they perceived to be “flawed” data collected. What was striking 
was the disconnect between their perceptions and that of the two MTCU informants, the 
latter of whom expressed confidence that the KPIs and performance funding have had 
a positive impact, given that the graduation and employment rates remain consistently 
high. The MTCU informants felt that the KPI requirement realized the government’s goal 
of encouraging university behaviour to support the government’s priorities. The Broad- 
hurst members also stated that the KPI initiative had had a positive impact, but only 
by increasing the awareness of the accountability of governing boards and perhaps by 
providing them with needed information. Ten out of 11 respondents concluded that the 
current three KPIs are not the best indicators to demonstrate accountability, which runs 
counter to the intent of the Broadhurst Task Force during their initial task of identify¬ 
ing performance indicators. In the follow-up interviews, key informants stated that the 
current KPIs are disconnected from the performance of the institution. In addition, they 
claimed that their generic nature does not work well when institutions are so diverse— 
e.g., with different sizes, mandates, program mixes, and student demographic groups. 

Indeed, respondents questioned the overall value of the current KPIs because of the 
unreliability of data, including the method of calculation and the response rates. More¬ 
over, it was impossible to identify under/over-performance, as the results across the sys¬ 
tem are close to the mean. Burke and Minassians (2003) pointed out that information 
can be powerful if it is used appropriately. When these performance indicators aligned 
with performance funding, there was certainly an expectation of improvement or change 
in performance. However, universities were not able to improve the values for these indi¬ 
cators, as the relevant factors were not under their control. To address this “disconnect,” 
some university respondents suggested replacing “the graduation rates methodology with 
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Consortium for Student Retention Data Exchange (CSRDE) methodology to ensure in¬ 
ter-institutional consistency.” In the CSRDE methodology, students of the same cohort 
are tracked from their first year to their graduation year, whereas the MTCU calculates 
graduation rates using a single entering cohort of students and by determining whether 
or not they graduated within seven years. The MTCU methodology does not isolate stu¬ 
dents who transferred between universities and thus understates the graduation rates for 
students affiliated with the original university. The university respondents suggested that 
the KPIs should “certainly be re-assessed and/or recalculated” and be disassociated from 
performance funding, due to the expressed disconnect. 

In terms of other key performance indicators, the respondents recommended the use 
of the Common University Data Ontario indicators, student satisfaction, research, indica¬ 
tors identified in the Broadhurst Report, and institution-specific reporting. One respon¬ 
dent suggested that the Strategic Mandate Agreement (SMA) with the ministry was a 
more appropriate tool, as it respects university autonomy while differentiating institu¬ 
tions by absolute measures. Respondents also recommended the indicators included in 
the Multi-Year Accountability Agreement (MYAA). Another suggestion was to define the 
general expectations at the system level. This would enable the development of key indi¬ 
cators consistent with these expectations and would include both a standard section and 
a variable section, with the latter capturing the uniqueness of the institutions. 

With respect to the impact on institutional planning, contrary to the perception of 
MTCU participants, most university key informants claimed that KPIs do not have a high 
impact on their institutional planning process. Most participating universities use gradu¬ 
ation rates for institutional planning but not employment rates and OSAP default rates. 
For internal purposes, all the participating universities use other indicators. 

In discussing performance indicators and the impact of performance funding, the uni¬ 
versities may desire to effect improvement or changes in performance in order to obtain 
more funding from the ministry. However, universities believe that they are not able to im¬ 
prove these indicators, as the contributing factors are not under their control. The univer¬ 
sity sector, in general, does not believe that KPIs are having much of the intended impact. 

From the ministry participants’ perspective, however, the perception was that since 
the graduation and employment rates have remained high across the system, they are 
having a positive impact on performance. The participants did not indicate whether these 
indices were high or not prior to the introduction of KPIs. Most university participants 
felt that the graduation rates had some but not much impact in improving performance 
at both institutional and program levels. About half of the university respondents agreed 
that employment rates were somewhat effective in improving institutional performance, 
but only two university participants felt that employment rates were effective in improv¬ 
ing performance at the program level. On the other hand, OSAP loan default rates were 
not used at all because factors such as employment opportunities, labour market, and the 
economy were beyond any given institution’s control. 

Clark et al. (2009) stated that one of the roles of MYAAs was to subsidize the limita¬ 
tions of KPIs in terms of institutional performance. However, the majority of university 
participants disagreed that the introduction of the MYAA process has altered either the 
role or the impact of KPIs in their universities. They felt that the MYAA and KPIs should 
be consolidated under one accountability framework. When the decision was made to re- 
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quire that KPI reporting be made public, the government’s intent was to provide potential 
students and parents with the basis to make informed decisions in the selection of insti¬ 
tutions and programs (MTCU, 1999). On the sector side, colleges and universities could 
use the data to demonstrate their achievements and improve their programs and services. 
However, both MTCU and the sector participants agreed that they did not use existing 
KPIs fully for active student recruitment. MTCU participants confirmed that currently the 
government also expects KPIs (i) to monitor progress in key areas at the institutional and 
system levels in order to assist with evaluating programs and institutions, with internal 
benchmarking, and with comparison of institutions, and (ii) to have sufficient integrity 
for the ministry to attach funding. The findings in this study suggest that the three KPIs 
implemented currently do not achieve these goals. These ministry expectations aligned 
with the desire of most of the university participants but differed from what they per¬ 
ceived to be the ministry’s expectations. The participants claimed that the government 
focused on simplistic measures as a surrogate accountability scale, expected to identify 
key issues of institutional performance, and expected the KPIs to highlight institution- 
specific problems and to reflect institutional accountability. 

In summary, the performance that the university respondents considered important 
is not captured by the current KPIs, and consequently they are not reliable indicators of 
performance for either accountability purposes or performance assessment. In contrast, 
the ministry participants perceived the high graduation and employment rates to have a 
positive impact on performance. The responses further demonstrate that funding alloca¬ 
tions attached to KPIs are minimal and do not influence performance. 

Implications for Practice 

In order to gain broad acceptance and demonstrate effectiveness and accuracy, per¬ 
formance indicators must be based on valid and reliable tools specifically designed to 
measure performance metrics over which universities can exert control. The results of 
this study demonstrate a need to revisit Ontario’s KPIs. The following discussion will of¬ 
fer several recommendations for specific KPI characteristics. 

Consolidate the MYAA and KPIs 

As universities are supported by public funding, the outcomes of university educa¬ 
tion service should be measureable and reported in an accountable way. If the reporting 
of performance is to be at all useful, there is a need to ensure the integrity and compre¬ 
hensiveness of the data collected to measure performance. Mutually agreed upon perfor¬ 
mance measures should assist in decision making for both the institutions and the gov¬ 
ernment. To this end, these performance measures should also allow institutions to build 
institution-specific indicators as recommended originally by the Broadhurst Task Force, 
to enable institutions to improve their performance. Participants in this study recom¬ 
mended the consolidation of the MYAA and KPIs under one accountability framework. 

Clarify the Relationship between Funding and KPIs 

There is a need to explicitly address the question of whether or not to continue per¬ 
formance funding. Furthermore, performance monitoring should allow institutions (i.e., 
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their governing bodies) to evaluate operational efficiency, effectiveness and cost effective¬ 
ness, and any reward funding should be sufficient to make it worthwhile for the institu¬ 
tions to implement appropriate changes. 

Adopt the CSRDE Methodology 

Of the current KPIs, the university participants viewed only graduation rates to be 
most helpful for institutional planning, student recruitment, and other purposes. The 
study participants saw them to be effective factors in somewhat improving both institu¬ 
tional and program performance in that context. However, participants pointed out that 
because the current methodology does not isolate the students who transferred between 
universities, graduation rates for the originating university are understated. Participants 
recommended the adoption of the CSRDE methodology (i.e., students of the same cohort 
are followed through from the first year to the year they graduate). 

Disseminate COU and MTCU Data to the Public 

Currently, only the KPI-related data in the Ontario Graduate Survey (i.e., employ¬ 
ment rates) are published. Some of the data that would be useful to students and parents 
are not published by the universities, COU, or MTCU but should be made publicly avail¬ 
able. For example, the proportion of respondents working part-time and the number of 
respondents attending further post-secondary education would be useful information for 
students and parents (OUAC, 2009). 

Implications for Public Policy 

The findings of this study have the potential to enable more effective, evidence-based 
decision making on public policy by the government and on institutional policies by the 
universities. In instituting the KPIs, originally the Ontario government attempted to in¬ 
directly influence the behaviour of the universities to support its socio-economic goals of 
increasing (i) student participation in post-secondary education and (ii) employment rate 
after graduation. The government’s ultimate goal in instituting the KPIs and additional 
accountability measures, such as the MYAA, was to build a highly qualified workforce and 
monitor how effectively the universities use the public support of scarce resources. How¬ 
ever, the goals and missions of the universities are broader than that. 

Government policy for post-secondary education has become increasingly more mar¬ 
ket-oriented because of the unstable economic climate, not only in Ontario but world¬ 
wide. The KPI model for accountability has been in place and unchanged since the 1990s. 
The government has implemented the MYAA and SMA as additional accountability mea¬ 
sures. The findings in this study suggest that the KPI model is not working; universities 
perceive it to have had only a minimal impact on their behaviour and consider the KPIs 
to be more of a “nuisance” than a driver for program change, which may be partly due to 
high barriers, such as faculty tenure. There is a need to update government policy in order 
to achieve accountability goals and align institutions more closely with the universities’ 
core missions, empowering them to impact positively both their own goals and those of 
the government. Also, the new policy framework must take into account diversity within 
the university sector. In addition, I recommend consolidating the current measures of ac¬ 
countability—KPIs, MYAA, and SMA—into one coherent accountability framework. 
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Limitations of the Study 

Several factors, such as difficulty in accessing information and the reluctance of uni¬ 
versity representatives to participate in the survey, have influenced this study’s results 
through the various stages of data collection and analysis. 

Methodological Assumptions and Limitations 

A major challenge was locating the members of the Broadhurst Task Force to identify 
the original intent of the KPIs. Related to this was the methodological limitation of col¬ 
lecting historical data based on informants’ recall of past activities. This was especially 
true when members of the Broadhurst Task Force were recounting what happened two 
decades ago. However, what they did recall must have been important issues if they re¬ 
membered them so clearly that many years later. Furthermore, since the participants in 
the survey and interviews had nothing to gain or lose, it is reasonable to assume that their 
responses reflected what they actually perceived or remembered the issues to have been. 

Scope 

This study explored the perceptions of key informants in 11 (55%) of the 20 Ontario 
universities who participated in this study with respect to the implementation and subse¬ 
quent impact of KPIs in the universities. The extent to which current KPIs were aligned 
with individual universities’ mission statements and other tools or mechanisms was also 
explored. Participation in the study was not by random sampling, and because of the con¬ 
siderable diversity among Ontario universities in terms of size, geographic location, pro¬ 
gram mix, and intensity of teaching and research, the findings are not generalizable to all 
Ontario universities. This study involved a voluntary sample (N = 11; 55%) of English lan¬ 
guage universities in Ontario, and the findings cannot be generalized to the other English- 
language universities, nor to the francophone university in Ontario, or to other universities 
in the rest of Canada or elsewhere. This study did not address existing research related to 
performance indicators for post-secondary institutions in other Canadian provinces (since 
under the Canadian constitution, education is a provincial matter) or in international ju¬ 
risdictions, which may be similar to or very different from the Ontario experience. 

Key Informants 

The invited participants/respondents in this study were personnel directly involved 
in the areas of institutional planning, budgeting, and institutional performance and pro¬ 
vided a perspective from an institutional level. On the other hand, the deans of faculties, 
who are probably most directly involved in directing program changes, were not included 
and may well have had a very different viewpoint. 

Recommendations for Further Research 

This study confined itself to the evaluation of the KPI model more than 10 years after 
its implementation in Ontario, based on feedback from key informants in 11 of 20 public 
universities in Ontario. Further research in the following areas would build on these data 
to address the issue of having valid and reliable measures of accountability in publicly 
funded educational institutions: 
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1. What are the common elements and differences in the indicators that all of the 
20 universities currently use for different purposes? According to the findings, for 
internal purposes such as gauging student satisfaction, retention, accessibility, re¬ 
search, and time to completion, all the participating universities use other indi¬ 
cators. The participants mentioned that these institution-specific indicators are 
helpful for different purposes, such as managing performance and planning. They 
are also applicable to outcomes, outputs, and impact, and for evaluation and com¬ 
munications. A thorough analysis of the commonalities and differences among the 
indicators that the university sector employs would be helpful for all universities. 
It would be especially productive for the institutional planning offices to share best 
practices, identity the norms or standards of different programs, evaluate success, 
and encourage improvement. 

2. How and when do the institutions actually base their decisions on their KPI results? 
This study has examined the usefulness and effectiveness of KPIs for certain areas, 
such as institutional planning, student recruitment, setting tuition fees for specific 
programs, and improving institutional performance. The findings vary among the 
three KPIs and in different areas. The MTCU participants thought that the univer¬ 
sities indeed paid attention to and used the KPI results at the institutional level, 
and that the universities were committed to transparency and accountability as 
represented by the reporting of the KPIs. However, the university respondents did 
not share that view. For instance, one university interviewee specifically mentioned 
that his/her institution’s short-term plans did not relate to the KPIs at all, because 
it took seven years to receive data on graduation rates. Given the different per¬ 
spectives of the MTCU and university participants, as reflected in the analyses of 
the findings, it was not clear exactly how and when the institutions actually based 
decisions on their KPI results. Further research on this topic is recommended in 
order to assist the ministry in evaluating the usefulness of each of the three KPIs. 

Implications for the Theory of Accountability as Used in the KPIs 

By the time the Ontario government instituted the KPIs, accountability had become the 
driving force in initiating the requirement for regular and public reporting by Ontario uni¬ 
versities. This priority was already evident in the formal title of the Broadhurst Task Force: 
The Task Force on University Accountability. Accountability is also a core focus of other 
similar approaches, such as Oxford University’s White Paper on University Governance 
(2006), which declares that accountability is the first and most fundamental principle of 
governance. The governing bodies of Ontario’s publicly funded universities have a fidu¬ 
ciary responsibility to ensure that their universities operate effectively and efficiently. They 
also have a responsibility to ensure that the interests of their stakeholders are well served 
and that the funds provided by the public purse fulfill their declared institutional missions. 

As Trow (1996) stated, sustaining or improving institutional performance is one of the 
functions of accountability. He also stressed that institutions need to be examined inter¬ 
nally and externally in order to sustain or raise their quality of performance. However, 
as this study has shown, there is a strong perception among many university administra¬ 
tors that the current KPIs fall well short of meeting this goal. Consequently, the province 
needs new indicators and processes that more accurately reflect performance concern- 
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ing both governance and accountability. Furthermore, Clark, Moran, Skolnik, and Trick 
(2009) demonstrated that accountability and quality are closely interconnected, remark¬ 
ing that demanding accountability without considering the quality of the performance is 
limiting. The findings in this study suggest that the initial interpretation of accountability 
as defined by the three required KPIs is too narrow to be practically useful. To address 
this failing, there is a need for a comprehensive review and redefinition of the theory of 
accountability as applied to the governing boards of Ontario’s universities. 

Conclusion 

This study addressed important issues arising from the implementation of KPIs for 
Ontario universities. KPIs provide data that affect core decision making among multiple 
stakeholders in higher education: students, parents, universities, government, and tax¬ 
payers. Hence, the efficacy of KPIs is either directly or indirectly relevant to all citizens 
of Ontario. The findings of the study highlight the questionable effectiveness of the KPIs 
as they currently exist and point to possible alternatives for measuring university per¬ 
formance. Although not generalizable, the results of this research also provide valuable 
insights that may assist in evidence-based decision making in the revision or creation of 
MTCU policies that affect both Ontario’s universities and their stakeholders.^ 
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